{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "# Comparing Trained and Evaluated Model Experiments Using Vertex AI Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3a0651225470",
    "tags": []
   },
   "source": [
    "## Overview\n",
    "\n",
    "As a Data Scientist, you probably start running model experiments locally on your notebook. Depending on the framework you use, you would need to track parameters, training time series and evaluation metrics. In this way, you would be able to explain the modelling approach you would choose. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f3021b2963a6"
   },
   "source": [
    "### Lab objectives\n",
    "\n",
    "In this notebook, you learn how to use Vertex AI Experiments to compare and evaluate model experiments.\n",
    "\n",
    "This notebook uses the following Google Cloud ML services and resources:\n",
    "\n",
    "- Vertex AI Workbench\n",
    "- Vertex AI Experiments\n",
    "\n",
    "The steps performed include:\n",
    "\n",
    "- Log the model parameters.\n",
    "- Log the loss and metrics on every epoch to TensorBoard.\n",
    "- Log the evaluation metrics.\n",
    "- Compare two experiments.\n",
    "\n",
    "in Vertex AI Experiment of a recurrent neural network (RNN) for sentiment analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "title:migration,new",
    "tags": []
   },
   "source": [
    "Each learning objective will correspond to a #TODO in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/comparing_local_trained_models.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Install additional packages\n",
    "\n",
    "Install additional package dependencies not installed in your notebook environment, such as TensorFlow, TensorFlow Datasets and Vertex AI SDK. Use the latest major GA version of each package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m  WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script tfds is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script tensorboard is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The scripts estimator_ckpt_converter, import_pb_to_tensorboard, saved_model_cli, tensorboard, tf_upgrade_v2, tflite_convert, toco and toco_from_protos are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tfx-bsl 1.9.0 requires google-api-python-client<2,>=1.7.11, but you have google-api-python-client 2.55.0 which is incompatible.\n",
      "tfx-bsl 1.9.0 requires pyarrow<6,>=1, but you have pyarrow 9.0.0 which is incompatible.\n",
      "tfx-bsl 1.9.0 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-transform 1.9.0 requires pyarrow<6,>=1, but you have pyarrow 9.0.0 which is incompatible.\n",
      "tensorflow-transform 1.9.0 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-serving-api 2.9.1 requires tensorflow<3,>=2.9.1, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow<2.7.0,>=2.6.0, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, but you have tensorflow-io-gcs-filesystem 0.26.0 which is incompatible.\n",
      "pandas-profiling 3.1.0 requires markupsafe~=2.0.1, but you have markupsafe 2.1.1 which is incompatible.\n",
      "apache-beam 2.40.0 requires dill<0.3.2,>=0.3.1.1, but you have dill 0.3.5.1 which is incompatible.\n",
      "apache-beam 2.40.0 requires pyarrow<8.0.0,>=0.15.1, but you have pyarrow 9.0.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m\u001b[33m  WARNING: The scripts pyrsa-decrypt, pyrsa-encrypt, pyrsa-keygen, pyrsa-priv2pub, pyrsa-sign and pyrsa-verify are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script normalizer is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script tb-gcp-uploader is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tfx-bsl 1.9.0 requires google-api-python-client<2,>=1.7.11, but you have google-api-python-client 2.55.0 which is incompatible.\n",
      "tfx-bsl 1.9.0 requires pyarrow<6,>=1, but you have pyarrow 9.0.0 which is incompatible.\n",
      "tfx-bsl 1.9.0 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-transform 1.9.0 requires pyarrow<6,>=1, but you have pyarrow 9.0.0 which is incompatible.\n",
      "tensorflow-transform 1.9.0 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-serving-api 2.9.1 requires tensorflow<3,>=2.9.1, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow<2.7.0,>=2.6.0, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, but you have tensorflow-io-gcs-filesystem 0.26.0 which is incompatible.\n",
      "pandas-profiling 3.1.0 requires markupsafe~=2.0.1, but you have markupsafe 2.1.1 which is incompatible.\n",
      "google-cloud-recommendations-ai 0.2.0 requires google-api-core[grpc]<2.0.0dev,>=1.22.2, but you have google-api-core 2.8.2 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 2.55.0 which is incompatible.\n",
      "apache-beam 2.40.0 requires dill<0.3.2,>=0.3.1.1, but you have dill 0.3.5.1 which is incompatible.\n",
      "apache-beam 2.40.0 requires pyarrow<8.0.0,>=0.15.1, but you have pyarrow 9.0.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# The Vertex AI Workbench Notebook product has specific requirements\n",
    "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
    "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
    "    \"/opt/deeplearning/metadata/env_version\"\n",
    ")\n",
    "\n",
    "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_WORKBENCH_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\"\n",
    "\n",
    "! pip3 install {USER_FLAG} --upgrade tensorflow==2.8.0 tensorflow_datasets==4.5.2 -q\n",
    "! pip3 install --user --force-reinstall 'google-cloud-aiplatform>=1.15' -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Please ignore the incompatibility error and warnings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EzrelQZ22IZj"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lWEdiXsJg0XY"
   },
   "source": [
    "## Before you begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "oM1iC_MfAts1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID:  qwiklabs-gcp-01-babac012e883\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "PROJECT_ID = \"\"\n",
    "\n",
    "# Get your Google Cloud project ID from gcloud\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJYoRfYng0XZ"
   },
   "source": [
    "Otherwise, set your project ID here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "riG_qUokg0XZ"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
    "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "WXaOgYQVYya8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "!gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "03d8d65b914d"
   },
   "source": [
    "#### Region\n",
    "\n",
    "You can also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook.  Below are regions supported for Vertex AI. It is recommended that you choose the region closest to you.\n",
    "\n",
    "- Americas: `us-central1`\n",
    "- Europe: `europe-west4`\n",
    "- Asia Pacific: `asia-east1`\n",
    "\n",
    "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
    "\n",
    "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "3281bedf6d3c"
   },
   "outputs": [],
   "source": [
    "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
    "\n",
    "if REGION == \"[your-region]\":\n",
    "    REGION = \"us-central1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "06571eb4063b"
   },
   "source": [
    "#### Timestamp\n",
    "\n",
    "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "697568e92bd6"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgPO1eR3CYjk"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "MzGDU7TWdts_"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
    "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "cf221059d072"
   },
   "outputs": [],
   "source": [
    "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
    "    BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
    "    BUCKET_URI = f\"gs://{BUCKET_NAME}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://qwiklabs-gcp-01-babac012e883aip-20220818154737/...\n"
     ]
    }
   ],
   "source": [
    "! gcloud storage buckets create --location=$REGION --project=$PROJECT_ID $BUCKET_URI"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ucvCsknMCims"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "vhOb7YnwClBb"
   },
   "outputs": [],
   "source": [
    "! gcloud storage ls --all-versions --long $BUCKET_URI"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "pRUOFELefqf1"
   },
   "outputs": [],
   "source": [
    "# General\n",
    "import logging\n",
    "\n",
    "logger = logging.getLogger(\"logger\")\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "\n",
    "# Vertex AI\n",
    "from google.cloud import aiplatform as vertex_ai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xAoqKXTsR2Zd"
   },
   "source": [
    "### Define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "hl6BCLNnXrSD"
   },
   "outputs": [],
   "source": [
    "# Training\n",
    "DATA_DIR = \"data\"\n",
    "LOG_DIR = \"logs\"\n",
    "\n",
    "# Experiments\n",
    "TASK = \"classification\"\n",
    "MODEL_TYPE = \"rnn\"\n",
    "EXPERIMENT_NAME = f\"{PROJECT_ID}-{TASK}-{MODEL_TYPE}-{TIMESTAMP}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "inR70nh38PeK"
   },
   "source": [
    "### Initialize Vertex AI SDK for Python\n",
    "\n",
    "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "Nz0nasrh8T3c"
   },
   "outputs": [],
   "source": [
    "vertex_ai.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tUH9-Vitix5K"
   },
   "source": [
    "### Create TensorBoard instance using Vertex AI TensorBoard\n",
    "\n",
    "You can upload your TensorBoard logs by first creating a TensorBoard instance.\n",
    "\n",
    "Learn more about [TensorBoard overview](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-overview)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "NBpfzfA9iv3j"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Tensorboard\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.tensorboard.tensorboard_resource:Creating Tensorboard\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Create Tensorboard backing LRO: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/operations/5087686935921557504\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.tensorboard.tensorboard_resource:Create Tensorboard backing LRO: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/operations/5087686935921557504\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorboard created. Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.tensorboard.tensorboard_resource:Tensorboard created. Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this Tensorboard in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.tensorboard.tensorboard_resource:To use this Tensorboard in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tb = aiplatform.Tensorboard('projects/745211089335/locations/us-central1/tensorboards/2410111895898947584')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.tensorboard.tensorboard_resource:tb = aiplatform.Tensorboard('projects/745211089335/locations/us-central1/tensorboards/2410111895898947584')\n"
     ]
    }
   ],
   "source": [
    "vertex_ai_tb = vertex_ai.Tensorboard.create()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "jotNH6scnV4S"
   },
   "outputs": [],
   "source": [
    "vertex_ai.init(experiment=EXPERIMENT_NAME, experiment_tensorboard=vertex_ai_tb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tensorboard"
   },
   "source": [
    "## Training with Vertex AI Experiments\n",
    "\n",
    "Vertex AI enables users to track the steps (for example, preprocessing, training) of an experiment run, and track inputs (for example, algorithm, parameters, datasets) and outputs (for example, models, checkpoints, metrics) of those steps. \n",
    "\n",
    "Below you have some example of how track experiments to train recurrent neural network for sentiment analysis. \n",
    "\n",
    "To simplify the code, here you have helper function to cover the following steps:\n",
    "\n",
    "- Collect training data\n",
    "- Create text encoder\n",
    "- Build a RNN as baseline model\n",
    "- Build a LSTM as challenger model\n",
    "- Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "VVyNpW9Uqiyt"
   },
   "outputs": [],
   "source": [
    "!mkdir -p -m 777 $DATA_DIR $LOG_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "cTZGTxAXYKAE"
   },
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "tfds.disable_progress_bar()\n",
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "# Helpers ----------------------------------------------------------------------\n",
    "def get_dataset(data_dir, buffer_size, batch_size):\n",
    "    \"\"\"\n",
    "    Returns a tf.data.Dataset object containing the training data\n",
    "    Returns:\n",
    "        tf.data.Dataset: A tf.data.Dataset object containing the training data\n",
    "        buffer_size: The buffer size for prefetch data\n",
    "        batch_size: The batch size of dataset\n",
    "    \"\"\"\n",
    "\n",
    "    dataset = tfds.load(\n",
    "        \"imdb_reviews\",\n",
    "        download=True,\n",
    "        data_dir=data_dir,\n",
    "        with_info=False,\n",
    "        as_supervised=True,\n",
    "    )\n",
    "\n",
    "    train_dataset, test_dataset = dataset[\"train\"], dataset[\"test\"]\n",
    "\n",
    "    train_dataset = (\n",
    "        train_dataset.shuffle(buffer_size).batch(batch_size).prefetch(tf.data.AUTOTUNE)\n",
    "    )\n",
    "    test_dataset = test_dataset.batch(batch_size).prefetch(tf.data.AUTOTUNE)\n",
    "    return train_dataset, test_dataset\n",
    "\n",
    "\n",
    "def get_encoder(train_dataset, vocab_size):\n",
    "    \"\"\"\n",
    "    Returns a TextVectorization object for the encoder\n",
    "    Args:\n",
    "        train_dataset: A tf.data.Dataset object containing the training data\n",
    "    Returns:\n",
    "        TextVectorization: A TextVectorization object for the encoder\n",
    "    \"\"\"\n",
    "\n",
    "    encoder = tf.keras.layers.TextVectorization(max_tokens=vocab_size)\n",
    "    encoder.adapt(\n",
    "        train_dataset.map(lambda text, label: text, num_parallel_calls=tf.data.AUTOTUNE)\n",
    "    )\n",
    "    return encoder\n",
    "\n",
    "\n",
    "def get_baseline_model(encoder, model_params):\n",
    "    \"\"\"\n",
    "    Returns a tf.keras.Model object for the model as baseline\n",
    "    Args:\n",
    "        encoder: A TextVectorization object for the encoder\n",
    "        model_params: A dictionary with model parameters\n",
    "    Returns:\n",
    "        tf.keras.Model: A tf.keras.Model object for the model\n",
    "    \"\"\"\n",
    "\n",
    "    model = tf.keras.Sequential()\n",
    "    model.add(encoder)\n",
    "    model.add(\n",
    "        tf.keras.layers.Embedding(\n",
    "            input_dim=len(encoder.get_vocabulary()), output_dim=64, mask_zero=True\n",
    "        )\n",
    "    )\n",
    "    model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)))\n",
    "    model.add(tf.keras.layers.Dense(64, activation=\"relu\"))\n",
    "    model.add(tf.keras.layers.Dense(1))\n",
    "    model.compile(\n",
    "        loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
    "        optimizer=tf.keras.optimizers.Adam(\n",
    "            learning_rate=model_params[\"learning_rate\"],\n",
    "            beta_1=model_params[\"beta_1\"],\n",
    "            beta_2=model_params[\"beta_2\"],\n",
    "            epsilon=model_params[\"epsilon\"],\n",
    "        ),\n",
    "        metrics=[\"accuracy\"],\n",
    "    )\n",
    "    return model\n",
    "\n",
    "\n",
    "def get_stacked_model(encoder, model_params):\n",
    "    \"\"\"\n",
    "    Returns a tf.keras.Model object for the model as challenger\n",
    "        Args:\n",
    "            encoder: A TextVectorization object for the encoder\n",
    "            model_params: A dictionary with model parameters\n",
    "        Returns:\n",
    "            tf.keras.Model: A tf.keras.Model object for the model\n",
    "    \"\"\"\n",
    "\n",
    "    model = tf.keras.Sequential()\n",
    "    model.add(encoder)\n",
    "    model.add(\n",
    "        tf.keras.layers.Embedding(\n",
    "            input_dim=len(encoder.get_vocabulary()), output_dim=64, mask_zero=True\n",
    "        )\n",
    "    )\n",
    "    model.add(\n",
    "        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True))\n",
    "    )\n",
    "    model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)))\n",
    "    model.add(tf.keras.layers.Dense(64, activation=\"relu\"))\n",
    "    model.add(tf.keras.layers.Dropout(0.5))\n",
    "    model.add(tf.keras.layers.Dense(1))\n",
    "    model.compile(\n",
    "        loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
    "        optimizer=tf.keras.optimizers.Adam(\n",
    "            learning_rate=model_params[\"learning_rate\"],\n",
    "            beta_1=model_params[\"beta_1\"],\n",
    "            beta_2=model_params[\"beta_2\"],\n",
    "            epsilon=model_params[\"epsilon\"],\n",
    "        ),\n",
    "        metrics=[\"accuracy\"],\n",
    "    )\n",
    "    return model\n",
    "\n",
    "\n",
    "def train(train_dataset, test_dataset, model, epochs, steps, log_dir=LOG_DIR):\n",
    "    \"\"\"\n",
    "    Trains the model\n",
    "    Args:\n",
    "        train_dataset: A tf.data.Dataset object containing the training data\n",
    "        test_dataset: A tf.data.Dataset object containing the testing data\n",
    "        model: A tf.keras.Model object for the model\n",
    "        epochs: The number of epochs\n",
    "        steps: The number of validation steps\n",
    "        log_dir: The location of tf training logs\n",
    "    \"\"\"\n",
    "    tensorboard = tf.keras.callbacks.TensorBoard(log_dir=log_dir)\n",
    "    history = model.fit(\n",
    "        train_dataset,\n",
    "        validation_data=test_dataset,\n",
    "        epochs=epochs,\n",
    "        validation_steps=steps,\n",
    "        callbacks=[tensorboard],\n",
    "    )\n",
    "    return history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Uz7MSBcwVHgc"
   },
   "source": [
    "### Define a baseline\n",
    "\n",
    "As part of the experimentation phase, you would like to define a baseline model to have a performance benchmark which it is represented by recurrent neural network (RNN) in this case. Then you try to improve the ML solution with more complex training configuration that produce better results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TM1xGPRkjocQ"
   },
   "source": [
    "#### Run experiment and evaluate experiment runs using `with` statement\n",
    "\n",
    "This step would takes **15 min** approx. to finish. And it covers the following steps:\n",
    "\n",
    "- Initialize an experiment run\n",
    "- Log the parameters associated to training data\n",
    "- Log the parameters of the encoder\n",
    "- Log the parameters of the model\n",
    "- Train the model\n",
    "- Log the metrics for each epochs\n",
    "- Log the overall training metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "zRp-gJWhPKf0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Initialize the experiment.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1 to Experiment: qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.metadata.experiment_resources:Associating projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1 to Experiment: qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n",
      "INFO:root:Get the training and testing datasets.\n",
      "INFO:absl:No config specified, defaulting to first: imdb_reviews/plain_text\n",
      "INFO:absl:Load pre-computed DatasetInfo (eg: splits, num examples,...) from GCS: imdb_reviews/plain_text/1.0.0\n",
      "INFO:absl:Load dataset info from /tmp/tmpsu0ftwictfds\n",
      "INFO:absl:Field info.release_notes from disk and from code do not match. Keeping the one from code.\n",
      "INFO:absl:Field info.config_name from disk and from code do not match. Keeping the one from code.\n",
      "INFO:absl:Field info.config_description from disk and from code do not match. Keeping the one from code.\n",
      "INFO:absl:Field info.citation from disk and from code do not match. Keeping the one from code.\n",
      "INFO:absl:Field info.splits from disk and from code do not match. Keeping the one from code.\n",
      "INFO:absl:Field info.supervised_keys from disk and from code do not match. Keeping the one from code.\n",
      "INFO:absl:Field info.module_name from disk and from code do not match. Keeping the one from code.\n",
      "INFO:absl:Generating dataset imdb_reviews (./data/imdb_reviews/plain_text/1.0.0)\n",
      "INFO:absl:Downloading http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz into data/downloads/ai.stanfor.edu_amaas_sentime_aclImdb_v1PaujRp-TxjBWz59jHXsMDm5WiexbxzaFQkEnXc3Tvo8.tar.gz.tmp.45fbfd7fd99d4c81a3f6e74e94567902...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDownloading and preparing dataset 80.23 MiB (download: 80.23 MiB, generated: Unknown size, total: 80.23 MiB) to ./data/imdb_reviews/plain_text/1.0.0...\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Done writing data/imdb_reviews/plain_text/1.0.0.incompleteRF0TE4/imdb_reviews-train.tfrecord*. Number of examples: 25000 (shards: [25000])\n",
      "INFO:absl:Done writing data/imdb_reviews/plain_text/1.0.0.incompleteRF0TE4/imdb_reviews-test.tfrecord*. Number of examples: 25000 (shards: [25000])\n",
      "INFO:absl:Done writing data/imdb_reviews/plain_text/1.0.0.incompleteRF0TE4/imdb_reviews-unsupervised.tfrecord*. Number of examples: 50000 (shards: [50000])\n",
      "INFO:absl:Constructing tf.data.Dataset imdb_reviews for split None, from ./data/imdb_reviews/plain_text/1.0.0\n",
      "2022-08-18 15:52:02.495965: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64\n",
      "2022-08-18 15:52:02.496016: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-08-18 15:52:02.496042: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (tensorflow-2-6-20220818-205109): /proc/driver/nvidia/version does not exist\n",
      "2022-08-18 15:52:02.496588: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDataset imdb_reviews downloaded and prepared to ./data/imdb_reviews/plain_text/1.0.0. Subsequent calls will reuse this data.\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Get the encoder.\n",
      "INFO:root:Get the model.\n",
      "INFO:root:Train the model.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "391/391 [==============================] - 399s 1000ms/step - loss: 0.6305 - accuracy: 0.5826 - val_loss: 0.4679 - val_accuracy: 0.7414\n",
      "Epoch 2/2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-18 15:58:48.255852: W tensorflow/core/kernels/data/cache_dataset_ops.cc:768] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset  will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "391/391 [==============================] - 390s 997ms/step - loss: 0.3790 - accuracy: 0.8296 - val_loss: 0.3695 - val_accuracy: 0.8164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-18 16:05:18.400660: W tensorflow/core/kernels/data/cache_dataset_ops.cc:768] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset  will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.\n",
      "INFO:root:Evaluate model.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "391/391 [==============================] - 92s 234ms/step - loss: 0.3467 - accuracy: 0.8319\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Exit the run.\n"
     ]
    }
   ],
   "source": [
    "# Experiment Settings ----------------------------------------------------------\n",
    "RUN_ID_1 = \"run-1\"\n",
    "BUFFER_SIZE = 10000\n",
    "BATCH_SIZE = 64\n",
    "VOCAB_SIZE = 1000\n",
    "EPOCHS = 2\n",
    "STEPS = 20\n",
    "ROLE = \"baseline\"\n",
    "LR = 1e-4\n",
    "B_1 = 0.9\n",
    "B_2 = 0.999\n",
    "EPS = 1e-07\n",
    "\n",
    "# Initialize the experiment\n",
    "logging.info(\"Initialize the experiment.\")\n",
    "with vertex_ai.start_run(RUN_ID_1) as run:\n",
    "\n",
    "    # Get the training and testing datasets\n",
    "    logging.info(\"Get the training and testing datasets.\")\n",
    "    data_params = {\"buffer_size\": BUFFER_SIZE, \"batch_size\": BATCH_SIZE}\n",
    "    train_dataset, test_dataset = get_dataset(\n",
    "        \"./data\",\n",
    "        buffer_size=data_params[\"buffer_size\"],\n",
    "        batch_size=data_params[\"batch_size\"],\n",
    "    )\n",
    "    run.log_params(data_params)\n",
    "\n",
    "    # Get the encoder\n",
    "    logging.info(\"Get the encoder.\")\n",
    "    encoder_params = {\"vocab_size\": VOCAB_SIZE}\n",
    "    encoder = get_encoder(\n",
    "        train_dataset=train_dataset, vocab_size=encoder_params[\"vocab_size\"]\n",
    "    )\n",
    "    run.log_params(encoder_params)\n",
    "\n",
    "    # TODO 1: Get the model\n",
    "    logging.info(\"Get the model.\")\n",
    "    run.log_params({\"role\": ROLE})\n",
    "    model_params = # Your code goes here\n",
    "    model = get_baseline_model(encoder=encoder, model_params=model_params)\n",
    "    run.log_params(model_params)\n",
    "\n",
    "    # TODO 2: Train the model\n",
    "    logging.info(\"Train the model.\")\n",
    "    history = train(\n",
    "        # Your code goes here\n",
    "    )\n",
    "\n",
    "    run.log_params(history.params)\n",
    "    for idx in range(0, history.params[\"epochs\"]):\n",
    "        run.log_time_series_metrics(\n",
    "            {\n",
    "                \"train_loss\": history.history[\"loss\"][idx],\n",
    "                \"train_accuracy\": history.history[\"accuracy\"][idx],\n",
    "            }\n",
    "        )\n",
    "\n",
    "    # Evaluate model\n",
    "    logging.info(\"Evaluate model.\")\n",
    "    test_loss, test_accuracy = model.evaluate(test_dataset)\n",
    "    run.log_metrics({\"test_loss\": test_loss, \"test_accuracy\": test_accuracy})\n",
    "\n",
    "    # Exit the experiment\n",
    "    logging.info(\"Exit the run.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f8bd7ea02120"
   },
   "source": [
    "#### Evaluate the baseline model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "xmaumLrSqo1e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Get experiment status.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>experiment_name</th>\n",
       "      <td>qwiklabs-gcp-01-babac012e883-classification-rn...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run_name</th>\n",
       "      <td>run-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run_type</th>\n",
       "      <td>system.ExperimentRun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <td>COMPLETE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.role</th>\n",
       "      <td>baseline</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.learning_rate</th>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.steps</th>\n",
       "      <td>391.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.vocab_size</th>\n",
       "      <td>1000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.verbose</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.beta_2</th>\n",
       "      <td>0.999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.epochs</th>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.beta_1</th>\n",
       "      <td>0.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.batch_size</th>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.buffer_size</th>\n",
       "      <td>10000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.epsilon</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.test_loss</th>\n",
       "      <td>0.34672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.test_accuracy</th>\n",
       "      <td>0.83188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_series_metric.train_accuracy</th>\n",
       "      <td>0.82956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_series_metric.train_loss</th>\n",
       "      <td>0.378996</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                   0\n",
       "experiment_name                    qwiklabs-gcp-01-babac012e883-classification-rn...\n",
       "run_name                                                                       run-1\n",
       "run_type                                                        system.ExperimentRun\n",
       "state                                                                       COMPLETE\n",
       "param.role                                                                  baseline\n",
       "param.learning_rate                                                           0.0001\n",
       "param.steps                                                                    391.0\n",
       "param.vocab_size                                                              1000.0\n",
       "param.verbose                                                                    1.0\n",
       "param.beta_2                                                                   0.999\n",
       "param.epochs                                                                     2.0\n",
       "param.beta_1                                                                     0.9\n",
       "param.batch_size                                                                64.0\n",
       "param.buffer_size                                                            10000.0\n",
       "param.epsilon                                                                    0.0\n",
       "metric.test_loss                                                             0.34672\n",
       "metric.test_accuracy                                                         0.83188\n",
       "time_series_metric.train_accuracy                                            0.82956\n",
       "time_series_metric.train_loss                                               0.378996"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get experiment\n",
    "logging.info(\"Get experiment status.\")\n",
    "experiment_df = vertex_ai.get_experiment_df()\n",
    "experiment_df.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "evxi4ROLqpc_"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Get time series metrics.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>step</th>\n",
       "      <th>wall_time</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>train_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-08-18 16:05:19.552000+00:00</td>\n",
       "      <td>0.58260</td>\n",
       "      <td>0.630458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2022-08-18 16:05:19.674000+00:00</td>\n",
       "      <td>0.82956</td>\n",
       "      <td>0.378996</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   step                        wall_time  train_accuracy  train_loss\n",
       "0     1 2022-08-18 16:05:19.552000+00:00         0.58260    0.630458\n",
       "1     2 2022-08-18 16:05:19.674000+00:00         0.82956    0.378996"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get time series metrics\n",
    "logging.info(\"Get time series metrics.\")\n",
    "ts_runs_df = run.get_time_series_data_frame()\n",
    "ts_runs_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eoGfiTobVSSB"
   },
   "source": [
    "### Make experimental improvement to the model\n",
    "\n",
    "As challenger model you add a LSTM layer to the baseline model. This step would takes **20 min** approx. to finish.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZjEww1ESZFTn"
   },
   "source": [
    "#### Run experiment and evaluate experiment runs with `experiment` instance\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "kV8DIuUXVXkC"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:logger:Initialize the experiment.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2 to Experiment: qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.metadata.experiment_resources:Associating projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2 to Experiment: qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n",
      "INFO:root:Get the model.\n",
      "INFO:root:Train the model.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "391/391 [==============================] - 681s 2s/step - loss: 0.6096 - accuracy: 0.6070 - val_loss: 0.5632 - val_accuracy: 0.7547\n",
      "Epoch 2/2\n",
      "391/391 [==============================] - 667s 2s/step - loss: 0.3809 - accuracy: 0.8335 - val_loss: 0.3894 - val_accuracy: 0.8227\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Evaluate model.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "391/391 [==============================] - 157s 402ms/step - loss: 0.3583 - accuracy: 0.8334\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Exit the experiment.\n"
     ]
    }
   ],
   "source": [
    "# Experiment Settings ----------------------------------------------------------\n",
    "RUN_ID_2 = \"run-2\"\n",
    "ROLE = \"stacked\"\n",
    "\n",
    "# Initialize the experiment\n",
    "logger.info(\"Initialize the experiment.\")\n",
    "vertex_ai.start_run(RUN_ID_2)\n",
    "\n",
    "# Get the model\n",
    "logging.info(\"Get the model.\")\n",
    "run.log_params({\"role\": ROLE})\n",
    "model = get_stacked_model(encoder=encoder, model_params=model_params)\n",
    "vertex_ai.log_params(model_params)\n",
    "\n",
    "# Train the model\n",
    "logging.info(\"Train the model.\")\n",
    "history = train(\n",
    "    model=model,\n",
    "    train_dataset=train_dataset,\n",
    "    test_dataset=test_dataset,\n",
    "    epochs=EPOCHS,\n",
    "    steps=STEPS,\n",
    ")\n",
    "vertex_ai.log_params(history.params)\n",
    "for idx in range(0, history.params[\"epochs\"]):\n",
    "    vertex_ai.log_time_series_metrics(\n",
    "        {\n",
    "            \"train_loss\": history.history[\"loss\"][idx],\n",
    "            \"train_accuracy\": history.history[\"accuracy\"][idx],\n",
    "        }\n",
    "    )\n",
    "\n",
    "# TODO 3: Evaluate model\n",
    "logging.info(\"Evaluate model.\")\n",
    "test_loss, test_accuracy = # Your code goes here\n",
    "vertex_ai.log_metrics({\"test_loss\": test_loss, \"test_accuracy\": test_accuracy})\n",
    "\n",
    "# Exit the experiment\n",
    "logging.info(\"Exit the experiment.\")\n",
    "vertex_ai.end_run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ceaf7b22157d"
   },
   "source": [
    "#### Compare the baseline with the new model implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "bUYoJXexrpmu"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Get experiment status.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>experiment_name</th>\n",
       "      <td>qwiklabs-gcp-01-babac012e883-classification-rn...</td>\n",
       "      <td>qwiklabs-gcp-01-babac012e883-classification-rn...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run_name</th>\n",
       "      <td>run-2</td>\n",
       "      <td>run-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run_type</th>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.beta_2</th>\n",
       "      <td>0.999</td>\n",
       "      <td>0.999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.verbose</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.steps</th>\n",
       "      <td>391.0</td>\n",
       "      <td>391.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.epsilon</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.learning_rate</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.beta_1</th>\n",
       "      <td>0.9</td>\n",
       "      <td>0.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.epochs</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.buffer_size</th>\n",
       "      <td>NaN</td>\n",
       "      <td>10000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.batch_size</th>\n",
       "      <td>NaN</td>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.vocab_size</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.role</th>\n",
       "      <td>NaN</td>\n",
       "      <td>stacked</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.test_accuracy</th>\n",
       "      <td>0.8334</td>\n",
       "      <td>0.83188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.test_loss</th>\n",
       "      <td>0.35829</td>\n",
       "      <td>0.34672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_series_metric.train_loss</th>\n",
       "      <td>0.380914</td>\n",
       "      <td>0.378996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time_series_metric.train_accuracy</th>\n",
       "      <td>0.83348</td>\n",
       "      <td>0.82956</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                   0  \\\n",
       "experiment_name                    qwiklabs-gcp-01-babac012e883-classification-rn...   \n",
       "run_name                                                                       run-2   \n",
       "run_type                                                        system.ExperimentRun   \n",
       "state                                                                       COMPLETE   \n",
       "param.beta_2                                                                   0.999   \n",
       "param.verbose                                                                    1.0   \n",
       "param.steps                                                                    391.0   \n",
       "param.epsilon                                                                    0.0   \n",
       "param.learning_rate                                                           0.0001   \n",
       "param.beta_1                                                                     0.9   \n",
       "param.epochs                                                                     2.0   \n",
       "param.buffer_size                                                                NaN   \n",
       "param.batch_size                                                                 NaN   \n",
       "param.vocab_size                                                                 NaN   \n",
       "param.role                                                                       NaN   \n",
       "metric.test_accuracy                                                          0.8334   \n",
       "metric.test_loss                                                             0.35829   \n",
       "time_series_metric.train_loss                                               0.380914   \n",
       "time_series_metric.train_accuracy                                            0.83348   \n",
       "\n",
       "                                                                                   1  \n",
       "experiment_name                    qwiklabs-gcp-01-babac012e883-classification-rn...  \n",
       "run_name                                                                       run-1  \n",
       "run_type                                                        system.ExperimentRun  \n",
       "state                                                                       COMPLETE  \n",
       "param.beta_2                                                                   0.999  \n",
       "param.verbose                                                                    1.0  \n",
       "param.steps                                                                    391.0  \n",
       "param.epsilon                                                                    0.0  \n",
       "param.learning_rate                                                           0.0001  \n",
       "param.beta_1                                                                     0.9  \n",
       "param.epochs                                                                     2.0  \n",
       "param.buffer_size                                                            10000.0  \n",
       "param.batch_size                                                                64.0  \n",
       "param.vocab_size                                                              1000.0  \n",
       "param.role                                                                   stacked  \n",
       "metric.test_accuracy                                                         0.83188  \n",
       "metric.test_loss                                                             0.34672  \n",
       "time_series_metric.train_loss                                               0.378996  \n",
       "time_series_metric.train_accuracy                                            0.82956  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO 4: Get experiment\n",
    "logging.info(\"Get experiment status.\")\n",
    "experiment_df = # Your code goes here\n",
    "experiment_df.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "fORBLdvLrqFU"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Get time series metrics.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>step</th>\n",
       "      <th>wall_time</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-08-18 17:11:29.356000+00:00</td>\n",
       "      <td>0.609591</td>\n",
       "      <td>0.60700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2022-08-18 17:11:29.634000+00:00</td>\n",
       "      <td>0.380914</td>\n",
       "      <td>0.83348</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   step                        wall_time  train_loss  train_accuracy\n",
       "0     1 2022-08-18 17:11:29.356000+00:00    0.609591         0.60700\n",
       "1     2 2022-08-18 17:11:29.634000+00:00    0.380914         0.83348"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get time series metrics\n",
    "exp_run = vertex_ai.ExperimentRun(RUN_ID_2, experiment=EXPERIMENT_NAME)\n",
    "logging.info(\"Get time series metrics.\")\n",
    "ts_runs_df = exp_run.get_time_series_data_frame()\n",
    "ts_runs_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the notebook.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "uGvJieN5BjPG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting TensorboardRun : projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting TensorboardRun : projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete TensorboardRun  backing LRO: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/operations/2539775446736699392\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete TensorboardRun  backing LRO: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/operations/2539775446736699392\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorboardRun deleted. . Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:TensorboardRun deleted. . Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Artifact : projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2-tb-run\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting Artifact : projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2-tb-run\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete Artifact  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2-tb-run/operations/162297055950143488\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Artifact  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2-tb-run/operations/162297055950143488\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Artifact deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2-tb-run\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Artifact deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2-tb-run\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Context : projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting Context : projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete Context  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2/operations/558191610693681152\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Context  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2/operations/558191610693681152\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Context deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting TensorboardRun : projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting TensorboardRun : projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete TensorboardRun  backing LRO: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/operations/4773983074377531392\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete TensorboardRun  backing LRO: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/operations/4773983074377531392\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorboardRun deleted. . Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:TensorboardRun deleted. . Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584/experiments/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/runs/run-1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Artifact : projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1-tb-run\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting Artifact : projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1-tb-run\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete Artifact  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1-tb-run/operations/5169877629121069056\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Artifact  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1-tb-run/operations/5169877629121069056\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Artifact deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1-tb-run\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Artifact deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/artifacts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1-tb-run\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Context : projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting Context : projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete Context  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1/operations/747342795043241984\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Context  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1/operations/747342795043241984\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Context deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737-run-1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Context : projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting Context : projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete Context  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/operations/8759246532135354368\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Context  backing LRO: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737/operations/8759246532135354368\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Context deleted. . Resource name: projects/745211089335/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-01-babac012e883-classification-rnn-20220818154737\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Tensorboard : projects/745211089335/locations/us-central1/tensorboards/2410111895898947584\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting Tensorboard : projects/745211089335/locations/us-central1/tensorboards/2410111895898947584\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete Tensorboard  backing LRO: projects/745211089335/locations/us-central1/operations/4516855683152347136\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Tensorboard  backing LRO: projects/745211089335/locations/us-central1/operations/4516855683152347136\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorboard deleted. . Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Tensorboard deleted. . Resource name: projects/745211089335/locations/us-central1/tensorboards/2410111895898947584\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removing gs://qwiklabs-gcp-01-babac012e883aip-20220818154737/...\n"
     ]
    }
   ],
   "source": [
    "# Delete experiment\n",
    "exp = vertex_ai.Experiment(EXPERIMENT_NAME)\n",
    "exp.delete(delete_backing_tensorboard_runs=True)\n",
    "\n",
    "# Delete Tensorboard\n",
    "vertex_ai_tb.delete()\n",
    "\n",
    "# Delete Cloud Storage objects that were created\n",
    "delete_bucket = True\n",
    "\n",
    "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
    "    ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}\n",    "\n",
    "!rm -Rf $DATA_DIR $LOG_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "15fbfe47e022"
   },
   "outputs": [],
   "source": [
    "!rm -Rf $DATA_DIR $LOG_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "comparing_local_trained_models.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m95",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m95"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
